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COMPREHENSIVE OCCUPATIONAL DATA ANALYSIS PROGRAMS 
(CODAP): USE OF REXALL TO IDENTIFY DIVERGENT RATERS 


L. INTRODUCTION 


REXALL is a very powerful and flexible program within the Comprehensive Occupational Data 
Analysis Programs (CODAP) package, being designed primarily for analysing the inter-rater-agreement 
among judges’ task-factor ratings, One use of REXALL is for identifying divergent raters. After some 
general comments on REXALL, and a discussion of the term “divergent rater,” this report uses data from 
an actual study to elaborate on this use of REXALL. The decisions made on the basis of the REXALL 
output will then be verified by reference to the raw data. 


Il, GENERAL DISCUSSION AND BACKGROUND 


While pe enh was e designed primarily for handing task factor ratings, it may be used for analysing 
applied vee jidues nel sets of officer job descriptions on various attributes (Chiistal, 1975), and when a 
sample of officers rated a set of education profiles on educational suitability for service in a particular 


utilization field (Watson & Goody, 1975), It may be applied to rankings as well as to ratings. 


The items rated in the study used in this report were the 505 tasks in the task inventory for the 
Medical Service Specialist, A total of 93 first linc supervisors. rated the tasks on “Consequences of 


Inadequate Performance.” The instructions -page from the survey booklet is presented as Appendix A. It 


contains a definition of the task-factor involved, and describes the 9-point scale against which the judges. 
were to make their ratings. The 9-point scale also appears as Figure 1. These ratings were gathered in order . 
to obtain'a measure, the mean rating, of Consequences of Inadequate Performance for each task, REXALL 


- Was used to delermine whether any of the raters should be deleted fromthe study; and then to produce a =. 


card deck containing the mean task ratings, and to provide a measure of the inter-rater agreement on the 
ratings. 


Minimal 

Slight 

Not very serious 
Fairly serious 
Serious 

Very serious 
Extremely serious 
Almost disastrous 
Disastrous 


._ 8 8 


. 8 
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Figure 1, 9-point rating scale, 


A general description of REXALL and its relation to other CODAP programs has been documented 
by Christal and Weissmuller (1976). Figure 2 is a schematic presentation of the inputs to and outputs from 
the program. The het aad seein the items (tanks) that are eg are serine from nes 2. 


divergent raters and will be treated i in detail later i in + this report, 


Before discussing Table..1-in detail, the. term “divergent rater” needs clarification. This is simply a 


rater whose ratings are substantially different from those of the other raters. The most common of these are 
‘.. the non-cooperative raters who do not.even try to follow the instructions, generating instead some arbitrary 


pattern of responses. Another. type of awerment rater inverts the rating scale—instead of rating from low to 


RAW 
RATINGS 


CONTROL CARDS 


Each rater is 
identified by 
a Case ID, 


Specify options selected, 
the input format and the 
output headings. 


TABLES DESCRIBING ITEMS RATED TABLES DESCRIBING THE RATERS 


Item means, stand- ——— Frequency distrib- INTER-RATER _ RATER CORRELATION . 
“ard deviations and utions for item RELIABILITY TABLE ee TABLE 
number of raters. means and standard 
- (May also be punch= ' deviations. 


‘-ed as card decks.) 


Figure 2, REXALL inputs and outputs. 
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Table J, Rater Correlation Table (In Original Input Order) 


Rater N Sample 


ID _sCorrelation Ratings Mean 8D Mean  _T-Value 
I 
' Ol +670 505 3.92 1.00 5.12 20.22 
02 gel7 446 5.52 1,93 5.12 16.53 
03 .690 504 5.10 Lolo 5.12 21.35 
04 654 484 7.05 1,19 5.11 18.98 
05 486 503 4.21 98 5.12 12.45 
06 589 504 7.54 1.47 5.12 16.34 
“07 667 505 5.62 1.61 5.12 20.05 
08 . -664 505 5.00 1,11 5.12 19.91 
09 «103 505 Seis 1,62 5.12 22.18 
10. edal 505 §.10 2.36 5.12 _ 24.47 
11 -680 502 4.83 1.33 5.12 20.74 
12 612 504 6.35 1.35 5.12 17.35 
13 648 505 5.18 1.26 5.12 19,.10° * 
14 662 496 §.83- 3,12 5.12 19.64 
15 733 503 4.29 1.58 5.12- 24.11 
16° 152 ~ 505. ~. ~+6.03 1.85 5.12 3.45 
ee Tae ~,561 504 5.20  2sl?’ © “Sel2.. c+. 15619 
18 over 504 | 5.18 1.84 ~~ 5.12 me ree 
Pear | Soee +544 | . 505 .. .5.10 1,68 Seid, 14,53 
20: 681 504 . 5.94 1.56: -- 5.12....+ 20,86. 
a 6 a ae 8 - 0 ie! ean Lake 
0 @ @ e. @ Q 6 
i) @ @ 6 9 @ a 
74 496 503 5.88 1,34 5.12 12.80 
75 «547 505 - 6.52 1.60 5.12 14,66 
76 voll 480 4.64 1.25 5.16 12.99 
77 474 - §03 5.41 1.01 Dale 12.05 
78 704 505 4.92 1,00 5.12 22.24- 
79 .553 500 -6.29 2.06 5.13 14.81 
80 .707 501 6.14 1.11 5.11 22.32 
81 «480 502 8.21 1.74 5.12 12.23 
82 otal 504 5.92 1.70 5.12 23.75 
83 -523 150 “1.99 2,00 4.66 7.47 
84 474 95 4.71 71 ~ 4.87 5.19 
“85 -,115 505 1.02 15 5.12 ~- 2.61 
“86 .367 294 2.61 1.66 5.19 11.77 
87 .000 505 9.00 00 5.12 00 
88 .083 96 6.77 1.39 4,56 81 
89 . 708 505 6.72 2.06 5.12, 22.50 
90 +705. 503 4.73 89 5.12 22325 
91 518 503 ° 4.50 -64 5.12 13.55 
92 562 504 3.87 .69 5.12 15.24 
' 3.10 1.08 - 5.12 14.80 
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high the rater rates from high to low. As will be demonstrated shortly, both types are cavily identified from 
REXALL’s Rater Correlation Table. The non-cooperative rater Is usually dropped frou the study, and 


HI. USING THE RATER CORRELATION TABLE 
TO IDENTIFY DIVERGENT RATERS 


The Rater Correlation Table produced by REXALL contains a row of data for each rater, printed in 
the same order in which the cases were input to the program. Table | is an extract from the table for the 
iNustrative study being used i tfiiscre pqrt, raters 21 through 73 being omitted because af space limitations. 
The correlation column contains the correlations between cach rater’s ratings and the mitans. for all raters 
on the same tasks, the extreme right-hand column being a “T Value” for determining the level of 
significance of deviation of this correlation coefficient from zero, The other four colurng ore the number 
of items (tasks) rated by cach rater, the rater’s mean and standard deviation and the “sample mean.”* The 
sample mean is the mean of all raters’ ratings on the tasks rated by that rater, 


It is this table that is used to decide which raters are divergent and whether or not they should be - 


deleted from the study. The following paragraphs describe how these decisions were estabillsimdiin the study — 
being used to illustrate this report, For ease of explanation, the table entries containing the 20 highest and .” 
the 20 lowest correlations have been extracted from the original table and are presented 8 Table 2. When 

the program Is next revised, it is planned to provide an option to repeat fable 1, rearranged in descending. 


_orderof correlation coefficient, and also on other Variables contained In it. ; 


From Table 2 it is evident that the majority. of the correlation coefficients exceed 50 (in fact, 77%. 
exceed .50). Even the “good” raters vary in number of tasks rated and in mean and standard deviation of 
ratings. However, their high correlations with the group indicate thal they share similar perceptions of the 
relative consequences of inadequate performance of the tasks in the inventory. 


The prime indicators of divergent raters are the correlation coefficients, A high negative coefficient 
indicates inverted ratings, while an insignificant coefficient is the result of a-non-<ooperative rater, The 
raters who have made a genuine attempt to obey the instructions, but whose perceptions of the tasks differ 
from those of the majority of the raters. should have a relatively low, but significant, correlation 
coefficient. Unfortunately, this condition may also typify the rater who cooperates for part of the 
inventory and then becomes non-cooperative. 

Table 2 will now be used to identify the divergent raters in this study, There does not appear to be 
any cases of inverted ratings. The last six raters (in Table 2) are apparently non-cooperative raters, and at 
least the next two require special consideration as their perceptions of the tasks seem to be different from 
those of the majority of the raters, Each of the last eight raters in Table 2 will now he discussed 
individually. 

Rater 85, The correlation coefficient (—.115) cowd, in isolation, suggest inverted ratings but the 
other data available suggests otherwise, All tasks were rated. ‘The rater’s mean of 1.02 and standard 
deviation of .15 suggest that almost every task has been rated 1, with only a few higher ratings, Probably by 
chance, these higher ratings were allocated to lower than average tasks, giving rise ta the negative 
correlation coefficient. Rater 85 appears to have been non-coope rative and was deleted from the study. 


Rater 4]. A mean of 1.33 (standard deviation 81) indicates most tasks were rated | with: sprinkling 
of higher ratings, perhaps just to break up the pattern, Rater 41 seems to be another non-cooperative rater 
and was also dropped from the study. . 

Rater 87. The mean of 9.00 with a zero standard deviation indicates this rater rated every task 9, This 
is perhaps the most obvious type of lack of cooperation. Rater 87 was deleted. 


10 
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Rater N Sample 


8D Mean T-Value 

67 764 505 5.07 1.43 5.12 26.54 
10 737 505 3-10 . 2.36 5.12 24.47 
15 733 503 4,29 1.58 5.12 24,11 
82 727 504 5.92 1.70 5.12 23.75 
18 727 504 5.18 1,84 5.12 23.75 
35 726 503 4,49 1.44 5.12 23.66 
44 720 503 4.81 1.52 5.12 “23.23 
65 +720 494 4,19 1,07 5.12 23.00 
89 . 708 505 6.72 2.06 5.12 22.50 
80 707 501 6,14 1.11 5.11 22.32 
90 705 503 4,73 89 5.12 22.25 
78 704. 505 4.92 1,00 5.12 22,24 
09 703 505 5.13 1,62 5,12 22.18 
21 » 703 502 4,86 1,29 5.12 22,10 
60 699 479 4.4] 1.40 5.14 21.35 : 
72 -695 505 4.04 +84 5,12 21.69 
34 694 503 4,75 1.13 5.11 21,60 
03 690 504 5.10 1.13 5.12 21,35 
64 : 689 375 4,72 2,31 5.23 18,34 
53 » 687 505 6.13 1.70 5,12 21.18 
t ) @ e a | @ @ 
t Q t @ 0 ® 0 
@ @ t] a t ] } 6 
74 496 503 5,88 1.34 5.12 12.80 
05 - 486 503 4.21 .98 5.12 12.45 
81 + 480 502 8,21 1.74 5,12 12,23 
77 «474 503 5.41 1.01 5.12 12.05 
84 474 95 . 4 TL 71 4.87 5,19 
31 +454 479 8.44 1.14 5.14 11.12 
27 439 504 5.50. 2,06 5,12 10.96 
24 . 396 502 3,62 1.25 5,12 9.64 
52 377. 505 2.41 2.48 5.12 9.12 
48 358 505 4,98 194 5.12 8.60 
68 327 505 3,93 Pee | 5.12 7.76 
63 245 384 3.02 2.64 5.09 4.94 
54 194 504 7.79 2,03 5.12 4.44 
16 152 505 6,03 1,85 5.12 3.45 
61 106 505 5.14 96 5.12 2.38 
88 083 96 6.77 1,39 4.56 -81 
22 047 505 2,69 2,09 5.12 1.07 
87 000 505 9.00 290 5,12 00 
41 -.012 505 1.33 81 5.12 - .26 

} . 5,12 - 2.61 
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Rater 22, This combination of very low correlation (.047), low mean (2,69) and hight standard 
deviation (2,09), when the rater has responded to every task, stiygests a propensity of | or 2 ratings, witha 
nuinber of higher ratings (some up near the top of the scale) vllocaled indiscriminately among the other 
tasks. Rater 22 was treated us a non-cooperative rater gnd deleted. 


Rater 88, This rater responded to only 19% of the tasks. Although this 19% of the tasks tended to be 
less demanding than average (sample mean = 4.86; oveyal] mean = 5,12), this rateraveraged 6.77 onthem. 
The ratings that were provided are therefore very high, and a correlation of 083 suggests they are nut very 
realistic. The insigiticant correlation and low number of tasks rated justify deleting this rater on thre grounds 


of lack of cooperation, 


Rater 61. This case approaches the doubtful zone, The cortelatlon Is loo low to believe the rater has 
been entirely cooperative; and it is too low to assume u simple difference in per er nlion of the tasks, Rater 
61 was dropped from the study as being non-cooperative, 


Rater 16, Apatt from the relatively low correlation coefficient, this rater’s statistics seem fairly 
normal, except that the mean is perhaps a little high, There seer to be three possibilities: there is eithera 
genulne difference in perception, or a lack of discrimination power, or a lack of complete cooperation. A 
clear decision cannot be made on the data available. As there are plonty of ralers, and this one represents an 
isolated opinion if the ratings are genuine, the analyst chose to drop him from the study even though 
retention would not have seriously affected the task means and the inter-rater reli ability statistics, 


Rater $4, The analysis for this rater is very paratie| to that for Rater 16, However, the very high mean: 
(7.79) and standard deviation (2.03) indicates a very laste number of 9 ratings, perhaps genuinely believed 
to be justified, This rating pattern has depleted the river's discrimination power at the high end of the scale 
and thus caused the relatively low correlation coefficient. The decision on Whether or not to delete this. *- 
rater should have little effect on the objectives of the study. While ane qually good. case could be. made for: 
deletion, the analyst chose to retain this rater in the study, ee: : 


The next six to ten raters above Rater 54 in Tabie 2-also have app reciably lower correlations than the os 
- majority of raters and could be classed as divergent yaters. However, their ratings are considered gemuine 8 
-“ and the lower correlations can usually be explained. For example, the analysis for Rater8! would parallel ov 
that for Rater 54; Raters 63 and 52 are the same except thal by favoring the low endof the scale they have 
depleted their discrimination at that end of the seule, Raters 68, 48, and 84. demonsirate their low 
discrimination power by their lack of variation in ratings. As implied by the analysis for Rater 54, more 
than seven raters could have been deleted without significantly affecting the objectives of the study céther_ 
way. Where to ‘draw the line” must remain, for the tige being, a subjective decision on the part of the 
analyst. One more objective approach being examined is to have the Prograin progressively climinate the 
most divergent raters, one at a time, until the interrater teliability statistic for the stability of the item 
(task) means (Ry) ceases to increase. 


At this point, one further comment on divergent taters should be made. As is seen from Tables | and 
2, there are considerable differences among the raters’ means and among thelr standard deviations, caused 
by different perceptions of the words used to describe the levels on the 9-point mting scale. However, the 
magnitude of the correlations for the cases retained in the study are satisfactory evidence that these raters 
shared sufficiently similar perceptions of the relative consequences of inadequate performance of the tasks 
in the inventory..As it is a measure of the relative consequences that is required, no rater was declared 
divergent on the grounds of a high or low mean. In fact, these differencesin means and standard deviations 
only add within task variance that is not justified when relative measures are beingsought, Acco ningly the 
standardization option discussed by Christal and Weissmuller (1976) was used in this study to rernove the 
between rater variance. : 


The effects of deleting the seven divergent raters is presented in Tuble 3. Deleting the divergent raters, 
the ones who did. not cooperate and those who perceived the relative values of the tasks differently , does 
appreciably increase the inter-rater reliability and ihetefore imcreases the stabllity of the task means. This 
applies whether or not the data are standardized. 
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Table 3. Effects of Dele tng Seven Divergnt Raten, 
5 . as Rye kt 7 
Raw matings, Odedetions = 3S 94 
Raw ratings, 7deJetions - 86° 19S 952° 
: ; : _ Standardized matings, Odeletions 93 313 976 


~~ Slandardited tings, 7deletions 86 355.979 


IV. VERIFICATION OF DECZSIOMS BY RETERENCE 
TO RAW DATA 


“Having made the above decisions bused om the REXAEL Oltput, a copy of the raw data foxeach rater 
~ was obtained to examine thir validity. Extracts of the mw ratings are prtseriied In Appendixes B,C,ind D- 


_ as blocks of taw ratings, each block comlairaing all the ritin.gs nude by one iter. Bach block conlaimseight . 
- tows of digits, 69 digitsin each of the first seven rows anc 22 In the Last row. This bs a lotall of 50S digiils,. 


“one for each task in the inventory. Each digit isa laskKrating by the xater io which the bloce fens, a0 
indicating the rater did not rate the task, The fart digiton the first. rows tke ralingfor Task |, the last on 


co. that row is Task 69, and'so on, the Last digit on the Last. xow Deirag the rating on Task $05. ~~ 
)-: Appendix B isthe ratings for three “good” ta ters, prowided fo2 comtparon puxposes, It would ape 


~ that the raters have considered and rated each dak Individually, This does mot mean, ofconre, thal thelr 


|» differences in perception, arad of course there will alwaysbe sont erzor watiartee. 


", -\Mon-cooperative, Each will: now be discissed im tum, reflecting on the diagnoses: previously riade fron. - 
» Table 2 (the REXALL outpu). fae Ge kee se, of ee. ak 
<2" Rater 85, As predicted, Rater 85° ated newly every task 1, The seven 2 ratingsandone3 riting were 
all allocated to tasks which. most ralers cormideted less conmequentail than average. For oxampLe, the oe 
task, that:-was'rated 3 was ‘*schedule leaves or pases.” Fhis ashatdly as conseque ttialas, for exarmple, tasks 
*. “involving tendering emergency treatrientlo apatient, which wert obviously cited lower by Rater 85. ° 
ce > Rater 41: The prediction was that movst of this raler’s rathgs: would bel witha sprinkling of higher. 

| s fatings, Notice the three9 ratings, Thesethree tasks wert: (2) phn records maintenance, (b) alreet physical 
~ exercise or conditioning programs, and (c) dizect preparalion ared neulntenarice of records oF reports. 
Considering the nurnber of tasksin the inventory that could result inthe Imrmediiate death of” apatient, it |s 


is fairly common among non-covupe nitive rates whoscen to bellevé no one will ever delect thelr Lark, of 
cooperation ifthe survey booklet appearslo Ihave beenhonestly completed, Fe 


> Talings are identical, Because of differences in’ experierace, Work Location, tte, there wild be sore oe: 


In contrast! with Appendix B, Appeslixe C is the raw. mtiregs €or the six raters assayed as being 


@ 


ERIC 


JA Fui'Toxt Provided by ERIC 


Rarer. 61, ‘The dalle of this rater’ ’s TaW “responses explains his iow conelition ‘The isthe fot the 


nome Ret 71 tasks are very consistent with ‘those ‘of the group.. However, at:that point the rater has become a 
ete non-cooperative: ‘and rated all but! one of the ternaining tasks 5. To examine how good this rater’s first 71 ; 
"-Tesponses Were, ‘the table: entries that would have resulted had tasks 72 through 505 not been rated at all, 


were computed. They appear as Table 4: These statistics a are quite acceptable. It would have’ been far better... 
if Rater 61 had left these. eenlaing tasks blank, eae nee 


- Table4; Rater Correlation Table Entries... 
for Rater Number 61 with 
Tasks 7 2-505 Treated as Blank 


= oe ; 


- N Ratings = 71 
».Mean = 6.03 
Standard Deviation = 2.34 
Sample Mean = 451 
T- Value = 6.87 


The 1 raw data ite the two bordesline cases (Raters: 16 and 54) are presented in Appendix D, Asse" 
_. suggested in the. earlier analysis, their. ratings . should appear..fairly normal. unless. ‘they. were partially. 
cooperative raters (like Rater 61). Except for the abnormally large number of high ratings, particularly for. 

' Rater 54, both rating patterns seem reasonable, As already mentioned, the decision on whether to accept or- 
“reject these two raters is rather subjective, either. course having little effect on the mean ratings of the tasks... 
Rater 54 has given more valid ratings than Rater 16, although they are far from perfect, There seems to be. -..- 
© tor reason to to slenye the previous decision which was to accept one e and reject the other, >. wee, ghee 


oo 
a 
4 


hi CONCLUSION 


oan the dedistaas regarding diveipact ree in this study, ae on the basis of REXALL's Bite 


Ss eueitien Table, have been verified by reference-to the raw data. There weye no cases of inverted ratings 
in this study, although arithmetical inversion of such ratings in another study conducted by the authordid 
testify to the validity of treating high negative correlations as indicators of scale reversal; The Interpretation... 

of relatively low, but significant, correlation coefficients must remain somewhat subjective for the time - 
>. - “being. More objective approaches to the handling. of such cases are currently under consideration by ne Air. 
a Fore Hunan Resources Laboratory. _— see Ses 
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: APPENDIX A. INSTRUCTIONS USED. BY. JUDGES “WHEN RATING TASKS. IN" 
MEDICAL SERVICES SPECIALIST INVENTORY ON CONSEOUENCES ~ 
OF acai aa PERFORMANCE 


Explanation 


This booklet contains a listing of tasks performed in your career 
ladder. You are asked to rate each task to indicate the Probable 
Consequences of Inadequate Performance of the task, In the Air Force,  ~ 
“the consequences of “inadequate performance of some tasks are much more 
serious than for other tasks. For example, if inadequate performance 
of a task will almost certainly cause an aircraft to: crash, or a 
warehouse to burn down, or an airman to die, this would be more serious 
than inadequate performance of a task which ‘mere ly causes inconvenience © 
and irritation. As another example, the probable consequences of 
inadequate performance in responding to a fire alarm would be much more 
serious than the probable consequences of inadequate performance in 
folding haspital linen. 


Definition 


‘Consequences of Inadequate Performance is a measure of the seriousness 
of the probable consequences of inadequate performance of a task, It is 
measured in terms of possible injury or death, wasted supplies, damaged 
equipment, wasted man-hours of work, etc. 


‘Your Task 


Using che rating scale below, assign a numerical rating to each cask 
in this booklet which you feel describes the probable consequences of 
inadequate performance of the task, Make your ratings by simply writing 
a number 1 chrough 9 in the column to the right of each task. Please 
ap eempe to rate all tasks. 


Rating Seale. 


Tf tlae task is not done correctly, the probable consequences of, 
aneceduate performance ¢ would be: 


|. Minimal (inadequate performance has minimal consequences) 


2... Slight 
3. Not very serious 
r) : 4. Fairly serious 


. Serious 

6. Very serious 

7, Extremely serious ; ie ge a, 
8. . Almost disastrous. . isa 
9. Disastrous (inadequate performance has disastrous consequences) 


Your efforts in completing this booklet will be sincerely appreciated, When:" 
ao have finished your ratings, please return this booklet to your CBPO/DPMPC. 


APPENDIX B, RAN RATINGS FOR THREE TYPICAL "GOOD" RATERS 


2 Rater Number 15 

4113222130345 424343 32432 33223 42 32224 333332322244 43314 334335 3342433334 
4333433334335 33433323343 4333332 233533324533 43334 434333333335 333343222 
426 3434433466645 5433333545 4466665 3333332366 3455555555 5365665455434667 
77777337776455 3444444556 5655445 66 77566 5555555556655554553343334333443 
54755 44365 485556779 44456 65 44443 33954654539465 554 34335 775444444444 3334 
444 4343223443343 32263344 2322346 6977757 7525456 569 84594549 749666906 46 35 
55445 443334533347 3364755 4754334 437666345555 533333354769 766 4559445 5665 
5965577545 556644993568 


Rater Number. 44 


2124555 3454355 14450043324552241 41 54445521 133335564 11445 3535 1253545515 
43555 4555353555555333233 3332222 1355345 2433353344566 333333333366 363343 
3375556544577 748446 4436756 64567 83 665 66 63445 75 5544655575 85445 35 5333555 
6575834555654443 334564665345 44665 65446 666575456777 746655335556 5665752 
4476333366 465656 76754356575 765443 854544568468 85 445 33 7666565566655 2265 
6653554444665555 33454455 45 4446767555 46 6455555 56 785 8845457577 36 8344345 
46555 455555635565 356556545 5554564 76667 555578777777 75 867 77536 78665 5588 
5955567855557757993357 


Rater Number 53 


3231324225435434374 1347246345256 35545 443343324 47316445 56573573636644._. 
5556547556527446 3554455684 45 44564 77 746 286666665 54765 5546555556 7555544 
65 86667666689 96855855 5665885 89993675555468955 77 669 76 95555555355655668 
7799957889 855 76766887 88887 88889 8888777 888787888877 77 887 8666 7778777776 

66977775885 866 7888777 85666 5888555 97567 787957878 77656 887677877777 74587 
7777885 556777767527 75 25856 33 468889885665 666 87 8898788877 89 7888897766 78 
787775666776 37788378887 768685 7785 86 665 76 5385766 78787859 7568559587 7887 
7977589545 344777785577 
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AITENDLA ©, SAW SBALINGS PUR OLA NUNS LUYPERALLIVE RALENO 


Rater Number 85 


199.191919191.19.191199122 121131191919991911190181121112119119717117111111 
17119991291911119799229019019199199199119199999019.19991191 1919919111111 
4990.99919090.19499990191191019199991901199999017011190909909919919099091 
19919999991919090199999990191999919991991919990 1019199191 1919919119111 
499.99199099.19.091199991799019199191999199119017719199991991919999999101 
414499099.99999999799990199990901999991999199197 1919197 1909919919799197 
4419949999919199919999010971919199199999999999170 1919999991 1799919199111 
941119119119117197171171 


Rater Number 41 


222171191991 41121122291 191111 1399111112119110191111711119111122322222222 
23219111141221919919911119112110191999991991999019191900901909000019191 
| 199.199990.90.19199190111112 2212222112234 1921111321222 111912222122212122 
1232211919119199149190190199191111117922971991 1119199191 11191112112122 
12321231222211112221221922221912111919911191990 1419197 19119199 191112212 
1411111222221331111321171111111323333222142 11122 1113333222111221 11111 
94999199999.199999999901901919919999999999999901919990191199917199919119 
1741999719717197191171 


Rater Number 87 


999 9999999999999999999999 9999999999999999999999999999 999 9999999599999 
99999999999 99999999999 999 9999999999999999999999999999 9999999999999999 
9999999999999999999999 999 9999999999999999999999999999 999 9999999999999 
99999999999 99999999999 999 9999999999999999999999999999 999 9999999999999 
99999999999 99999999999 999 99999999999999999999999 99999999 9999999999999 
99999999999 99999999999 999 999999999999999999999 9999999999 9999999999999 
99999999999 99999999999999 99999999999999999% 39999 99999 9999999999999999 
9999999999999999999999 
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Rater Number 22 


41527522147 1971871112251 1721611312273261 1162421111 1156666584475748324 
94447755 7475844477447566 2422222255 72642646421226212421112461111113142 
19171755555 5221222111111311211524557252271511171122112113112116111111 


41211191,1112115243331224 342535221311 13444214244 2555231111117541525112 


431451 1125152521 351411 112255555225 7411135541221 1572223222 191244421152 


Tugoqers 14110010 ena a dda Ta AT TTT Hata tt Taga azt TI 2T A143 TB T1 22 eas 


1111924411111111151111 


Rater Number 88 


656 75556 667655666 7877679 155565679876 75556667897 779989898889987 776 7679 


86789777 788 89 777666650556 7000090 7000000000000000000000000000000000000 
00000000 0000000000000000000000000000000000000000000000000000000000000 


000000000000000000000000000000000000000000000000000000000000000000000 


00000000 0000000000000000000000000000000000000000000000000000000000000 


00000000 0000000000000000000000000000000000000000000000000000000000000 ~ 


900000000000000000000000000000000000000000000000000000000000000000000 _ 
00000 00000000000000000 


Rater WMumber 61 


72 147543698499147449777247465546664879 797166 589999979489 6683692755529 aN 


94555555 5555555555555555555555555555555555555555555555555555555555555 
§§555555 55555555 55555555555555555559555555555555555555555555555555555 
§$5555555555555555555 555555555555 559 555555559555 5555555555555 555555555 | 
49959595 55555555 55595559555555555555955555595555555555555555555155555 


5§ 555 55555555555 5555555555 5555555595 555555555555555555555555555555555 
~ $555 55555555555555555 


APPENDIX D, RAW RATINGS. FOR, TWO BORDERLINE CASES 
Rater Number 16 


7578845569 189 849 77668874778666467886 75544467787 7864776868876 785878978 
8768877878689 87 7786 78865 87 899 98979 88997977 788889 89 9999899999 899999999 . 
9999899998989999 879 88899 89 889899 8898888777 89777779 88888 877585 77566577 


789 89 45668777 86544565 4656 335444 7566547665 586866887 7644454654557554763. 


5374653267 27336455 454656 46 466 76.5 3775564559578866 45 44887 545 74454456 456 
66755 74455 74555443454 355 35 33346 46 77 7556455674745 7555334644 75557443334 , 
5444434645 443443.43444345 46 46454643445 3554544344555 4455564565463754554_ 
47362456 34554334563455 


Rater Number 54 


9477955699969958897779979897995888599998059778989887799 8899797999999 
9559998899 99889 7999 76999 78955 1415557995 79999998799 94555 99986 699999498. . 
9912229966 69999999999 669999999999 7999999999999999999999 99999999 799999 
9999999999 99999999999 222559299999999997 795 7777779997777 17799 799997779. 
9999955599 999999999 7999999997599999999999 799999999 5999895999999995999 . 
- 9957777799979999 75599 896525299899999 79999992999999 5955959917593259945 
. 99997995 7118195999895 999 9992 72a sho aa Aa ae podnere ener 1nso7 eee lees, i 
56772249 22 299927999298 . 
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